In this project we will try to find an optimal location for a restaurant. Specifically, this report will be targeted to stakeholders interested in opening an Italian restaurant in New York, USA.
Since there are lots of restaurants in downtown Toronto we will try to detect locations that are not already crowded with restaurants. We are also particularly interested in areas with no Italian restaurants in vicinity. We would also prefer locations as close to city center as possible, assuming that first two conditions are met.
We will use our data science powers to generate a few most promissing neighborhoods based on this criteria. Advantages of each area will then be clearly expressed so that best possible final location can be chosen by stakeholders.
Based on definition of our problem, factors that will influence our decission are:
We decided to use regularly spaced grid of locations, centered around city center, to define our neighborhoods.
Following data sources will be needed to extract/generate the required information:
Let's create latitude & longitude coordinates for centroids of our candidate neighborhoods. We will create a grid of cells covering our area of interest which is aprox. 12x12 killometers centered around Times Square, New York, USA.
Let's first find the latitude & longitude of Times Square, New York, using specific, well known address and Open Cage geocoding API.
#@@hidden
import numpy as np # library to handle data in a vectorized manner
import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
import json # library to handle JSON files
#!conda install -c conda-forge geopy --yes # uncomment this line if you haven't completed the Foursquare API lab
from geopy.geocoders import Nominatim # convert an address into latitude and longitude values
import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe
# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors
# import k-means from clustering stage
from sklearn.cluster import KMeans
#!conda install -c conda-forge folium=0.5.0 --yes # uncomment this line if you haven't completed the Foursquare API lab
!pip install folium
import folium # map rendering library
!pip install geocoder
!pip install opencage
from opencage.geocoder import OpenCageGeocode
from pprint import pprint
OpenCage_api_key = 'b8eed9bf2e234e87922d5f91506eec0a'
geocoder = OpenCageGeocode(OpenCage_api_key)
print('Libraries imported.')
Collecting folium
Downloading https://files.pythonhosted.org/packages/72/ff/004bfe344150a064e558cb2aedeaa02ecbf75e60e148a55a9198f0c41765/folium-0.10.0-py2.py3-none-any.whl (91kB)
|████████████████████████████████| 92kB 13.4MB/s eta 0:00:01
Collecting branca>=0.3.0 (from folium)
Downloading https://files.pythonhosted.org/packages/63/36/1c93318e9653f4e414a2e0c3b98fc898b4970e939afeedeee6075dd3b703/branca-0.3.1-py3-none-any.whl
Requirement already satisfied: numpy in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (1.15.4)
Requirement already satisfied: requests in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (2.21.0)
Requirement already satisfied: jinja2>=2.9 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from folium) (2.10)
Requirement already satisfied: six in /opt/conda/envs/Python36/lib/python3.6/site-packages (from branca>=0.3.0->folium) (1.12.0)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (1.24.1)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->folium) (2019.6.16)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from jinja2>=2.9->folium) (1.1.0)
Installing collected packages: branca, folium
Successfully installed branca-0.3.1 folium-0.10.0
Collecting geocoder
Downloading https://files.pythonhosted.org/packages/4f/6b/13166c909ad2f2d76b929a4227c952630ebaf0d729f6317eb09cbceccbab/geocoder-1.38.1-py2.py3-none-any.whl (98kB)
|████████████████████████████████| 102kB 16.0MB/s ta 0:00:01
Requirement already satisfied: six in /opt/conda/envs/Python36/lib/python3.6/site-packages (from geocoder) (1.12.0)
Collecting ratelim (from geocoder)
Downloading https://files.pythonhosted.org/packages/f2/98/7e6d147fd16a10a5f821db6e25f192265d6ecca3d82957a4fdd592cad49c/ratelim-0.1.6-py2.py3-none-any.whl
Requirement already satisfied: requests in /opt/conda/envs/Python36/lib/python3.6/site-packages (from geocoder) (2.21.0)
Requirement already satisfied: future in /opt/conda/envs/Python36/lib/python3.6/site-packages (from geocoder) (0.17.1)
Requirement already satisfied: click in /opt/conda/envs/Python36/lib/python3.6/site-packages (from geocoder) (7.0)
Requirement already satisfied: decorator in /opt/conda/envs/Python36/lib/python3.6/site-packages (from ratelim->geocoder) (4.3.2)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->geocoder) (1.24.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->geocoder) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->geocoder) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests->geocoder) (2019.6.16)
Installing collected packages: ratelim, geocoder
Successfully installed geocoder-1.38.1 ratelim-0.1.6
Collecting opencage
Downloading https://files.pythonhosted.org/packages/6d/f2/ed48d7e2fbd06f0ac8dbd511fecc233b68b523daccaae9fb1e6e56b240d4/opencage-1.2-py3-none-any.whl
Requirement already satisfied: Requests>=2.2.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from opencage) (2.21.0)
Requirement already satisfied: six>=1.4.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from opencage) (1.12.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from Requests>=2.2.0->opencage) (2019.6.16)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from Requests>=2.2.0->opencage) (2.8)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from Requests>=2.2.0->opencage) (1.24.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from Requests>=2.2.0->opencage) (3.0.4)
Installing collected packages: opencage
Successfully installed opencage-1.2
Libraries imported.
import requests
import geocoder
def get_coordinates(api_key, address,verbose=False):
try:
url = 'https://api.opencagedata.com/geocode/v1/google-v3-json?key={}&address={}'.format(api_key, address)
response = requests.get(url).json()
#response = url
if verbose:
print('Open Cage API JSON result =>', response)
results = response['results']
geographical_data = results[0]['geometry']['location'] # get geographical coordinates
lat = geographical_data['lat']
lon = geographical_data['lng']
return [lat, lon]
except:
return [None, None]
address = 'Times Square,New York City, USA'
ny_city = get_coordinates(OpenCage_api_key,address)
print('Coordinate of {}: {}'.format(address, ny_city))
Coordinate of Times Square,New York City, USA: ['40.75728055', '-73.9858550354592']
Now let's create a grid of area candidates, equaly spaced, centered around city center and within ~6km from Times Square,New York City. Our neighborhoods will be defined as circular areas with a radius of 300 meters, so our neighborhood centers will be 600 meters apart.
To accurately calculate distances we need to create our grid of locations in Cartesian 2D coordinate system which allows us to calculate distances in meters (not in latitude/longitude degrees). Then we'll project those coordinates back to latitude/longitude degrees to be shown on Folium map. So let's create functions to convert between WGS84 spherical coordinate system (latitude/longitude degrees) and UTM Cartesian coordinate system (X/Y coordinates in meters).
!pip install shapely
import shapely.geometry
!pip install pyproj
import pyproj
import math
def lonlat_to_xy(lon, lat):
proj_latlon = pyproj.Proj(proj='latlong',datum='WGS84')
proj_xy = pyproj.Proj(proj="utm", zone=33, datum='WGS84')
xy = pyproj.transform(proj_latlon, proj_xy, lon, lat)
return xy[0], xy[1]
def xy_to_lonlat(x, y):
proj_latlon = pyproj.Proj(proj='latlong',datum='WGS84')
proj_xy = pyproj.Proj(proj="utm", zone=33, datum='WGS84')
lonlat = pyproj.transform(proj_xy, proj_latlon, x, y)
return lonlat[0], lonlat[1]
def calc_xy_distance(x1, y1, x2, y2):
dx = x2 - x1
dy = y2 - y1
return math.sqrt(dx*dx + dy*dy)
print('Coordinate transformation check')
print('-------------------------------')
print('ny_city longitude={}, latitude={}'.format(ny_city[1], ny_city[0]))
x, y = lonlat_to_xy(ny_city[1], ny_city[0])
print('ny_city UTM X={}, Y={}'.format(x, y))
lo, la = xy_to_lonlat(x, y)
print('ny_city longitude={}, latitude={}'.format(lo, la))
Collecting shapely
Downloading https://files.pythonhosted.org/packages/38/b6/b53f19062afd49bb5abd049aeed36f13bf8d57ef8f3fa07a5203531a0252/Shapely-1.6.4.post2-cp36-cp36m-manylinux1_x86_64.whl (1.5MB)
|████████████████████████████████| 1.5MB 8.9MB/s eta 0:00:01
Installing collected packages: shapely
Successfully installed shapely-1.6.4.post2
Collecting pyproj
Downloading https://files.pythonhosted.org/packages/e5/fd/eb99d24327e248a5e93cec65eedf22a751f70723384a832837eef1f80509/pyproj-2.2.2-cp36-cp36m-manylinux1_x86_64.whl (11.2MB)
|████████████████████████████████| 11.2MB 13.6MB/s eta 0:00:01
Installing collected packages: pyproj
Successfully installed pyproj-2.2.2
Coordinate transformation check
-------------------------------
ny_city longitude=-73.9858550354592, latitude=40.75728055
ny_city UTM X=-5814666.312630144, Y=9867206.203134084
ny_city longitude=-73.98585503545884, latitude=40.75728054999889
Let's create a hexagonal grid of cells: we offset every other row, and adjust vertical row spacing so that every cell center is equally distant from all it's neighbors.
ny_city_x, ny_city_y = lonlat_to_xy(ny_city[1], ny_city[0]) # City center in Cartesian coordinates
k = math.sqrt(3) / 2 # Vertical offset for hexagonal grid cells
x_min = ny_city_x - 6000
x_step = 900
y_min = ny_city_y - 6000 - (int(21/k)*k*900 - 12000)/2
y_step = 900 * k
latitudes = []
longitudes = []
distances_from_center = []
xs = []
ys = []
for i in range(0, int(21/k)):
y = y_min + i * y_step
x_offset = 900 if i%2==0 else 0
for j in range(0, 21):
x = x_min + j * x_step + x_offset
distance_from_center = calc_xy_distance(ny_city_x, ny_city_y, x, y)
if (distance_from_center <= 6001):
lon, lat = xy_to_lonlat(x, y)
latitudes.append(lat)
longitudes.append(lon)
distances_from_center.append(distance_from_center)
xs.append(x)
ys.append(y)
print(len(latitudes), 'candidate neighborhood centers generated.')
161 candidate neighborhood centers generated.
Let's visualize the data we have so far: city center location and candidate neighborhood centers:
#!pip install folium
import folium
map_ny_city = folium.Map(location=ny_city, zoom_start=13)
folium.Marker(ny_city, popup='Times Square').add_to(map_ny_city)
for lat, lon in zip(latitudes, longitudes):
folium.Circle([lat, lon], radius=300, color='red', fill=False).add_to(map_ny_city)
map_ny_city
OK, we now have the coordinates of centers of neighborhoods/areas to be evaluated, equally spaced (distance from every point to it's neighbors is exactly the same) and within ~6km from Times Square, NY.
Let's now use Open Cage API to get approximate addresses of those locations.
def get_address(api_key, latitude, longitude, verbose=False):
try:
url = 'https://api.opencagedata.com/geocode/v1/json?q={}+{}&key={}'.format(latitude, longitude,api_key,)
response = requests.get(url).json()
if verbose:
print('Open Cage API JSON result =>', response)
results = response['results']
address = results[0]['formatted']
return address
except:
return None
addr = get_address(OpenCage_api_key, 40.75728055, -73.9858550354592)
print('Reverse geocoding check')
print('-----------------------')
print('Address of [{}, {}] is: {}'.format(40.75728055, -73.9858550354592, addr))
Reverse geocoding check ----------------------- Address of [40.75728055, -73.9858550354592] is: Times Square, U.S. Armed Forces Recruiting Station, New York, NY 10036, United States of America
print('Obtaining location addresses: ', end='')
addresses = []
for lat, lon in zip(latitudes, longitudes):
address = get_address(OpenCage_api_key, lat,lon)
if address is None:
address = 'NO ADDRESS'
address = address.replace(', United States of America', '') # We don't need country part of address
address = address.replace(', United States', '') # We don't need country part of address
addresses.append(address)
print(' .', end='')
print(' done.')
Obtaining location addresses: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . done.
addresses[150:170]
['Hoboken Newport Walkway- Hudson River Waterfront Walkway, Hoboken, NJ 07030', 'Lincoln Harbor Yacht Club, Hudson River Waterfront Walkway, Weehawken, NJ 07086', 'NJ 495, Weehawken, NJ 07086', '169 Jefferson Street, Weehawken, NJ 07086', '185 42nd Street, Weehawken, NJ 07087', '617 Washington Street, Hoboken, NJ 07030', '1012 Bloomfield Street, Hoboken, NJ 07030', 'Park Avenue, Hoboken, NJ 07030', '291 19th Street, Weehawken, NJ 07087', '166 Oak Street, Union City, NJ 07086', 'Union City, NJ']
Looking good. Let's now place all this into a Pandas dataframe.
import pandas as pd
df_locations = pd.DataFrame({'Address': addresses,
'Latitude': latitudes,
'Longitude': longitudes,
'X': xs,
'Y': ys,
'Distance from center': distances_from_center})
df_locations.head(10)
| Address | Latitude | Longitude | X | Y | Distance from center | |
|---|---|---|---|---|---|---|
| 0 | 25-20 49th Avenue, New York, NY 11101 | 40.742248 | -73.944078 | -5.817066e+06 | 9.861750e+06 | 5960.494946 |
| 1 | Two Court Square, 23-21 44th Drive, New York, ... | 40.747552 | -73.943882 | -5.816166e+06 | 9.861750e+06 | 5658.400834 |
| 2 | Citi Bike - 21 St & Queens Plaza North, 21st S... | 40.752858 | -73.943686 | -5.815266e+06 | 9.861750e+06 | 5488.852339 |
| 3 | 38-36 10th Street, New York, NY 11101 | 40.758163 | -73.943490 | -5.814366e+06 | 9.861750e+06 | 5464.201680 |
| 4 | 36th Avenue, New York, NY 11106 | 40.763470 | -73.943294 | -5.813466e+06 | 9.861750e+06 | 5586.367335 |
| 5 | Octagon Tennis Courts, Road 10, New York, NY 1... | 40.768777 | -73.943097 | -5.812566e+06 | 9.861750e+06 | 5846.152581 |
| 6 | 43 Paidge Avenue, New York, NY 11222 | 40.737072 | -73.950312 | -5.817966e+06 | 9.862530e+06 | 5723.635209 |
| 7 | 11-39 50th Avenue, New York, NY 11101 | 40.742376 | -73.950118 | -5.817066e+06 | 9.862530e+06 | 5256.424640 |
| 8 | 45-02 11th Street, New York, NY 11101 | 40.747681 | -73.949923 | -5.816166e+06 | 9.862530e+06 | 4911.211663 |
| 9 | 8-05 43rd Avenue, New York, NY 11101 | 40.752986 | -73.949728 | -5.815266e+06 | 9.862530e+06 | 4714.870094 |
...and let's now save/persist this data into local file.
df_locations.to_pickle('./locations.pkl')
Now that we have our location candidates, let's use Foursquare API to get info on restaurants in each neighborhood.
We're interested in venues in 'food' category, but only those that are proper restaurants - coffe shops, pizza places, bakeries etc. are not direct competitors so we don't care about those. So we will include in out list only venues that have 'restaurant' in category name, and we'll make sure to detect and include all the subcategories of specific 'Italian restaurant' category, as we need info on Italian restaurants in the neighborhood.
Foursquare credentials are defined in hidden cell bellow.
#@@hidden
CLIENT_ID = 'NB3VKMAEDNNQ2O3M0PY5SCPLMH4XF5XYSCXIB2R41SG0GMZ0' # your Foursquare ID
CLIENT_SECRET = 'HSFFI2CDK13FPG0AY25ESOWTNOKCYVSYTX04JCRUCH0LQAJF' # your Foursquare Secret
VERSION = '20180604'
LIMIT = 30
print('Your credentails:')
print('CLIENT_ID: ' + CLIENT_ID)
print('CLIENT_SECRET:' + CLIENT_SECRET)
Your credentails: CLIENT_ID: NB3VKMAEDNNQ2O3M0PY5SCPLMH4XF5XYSCXIB2R41SG0GMZ0 CLIENT_SECRET:HSFFI2CDK13FPG0AY25ESOWTNOKCYVSYTX04JCRUCH0LQAJF
# Category IDs corresponding to Italian restaurants were taken from Foursquare web site (https://developer.foursquare.com/docs/resources/categories):
food_category = '4d4b7105d754a06374d81259' # 'Root' category for all food-related venues
italian_restaurant_categories = ['4bf58dd8d48988d110941735','55a5a1ebe4b013909087cbb6','55a5a1ebe4b013909087cb7c',
'55a5a1ebe4b013909087cba7','55a5a1ebe4b013909087cba1','55a5a1ebe4b013909087cba4',
'55a5a1ebe4b013909087cb95','55a5a1ebe4b013909087cb89','55a5a1ebe4b013909087cb9b',
'55a5a1ebe4b013909087cb98','55a5a1ebe4b013909087cbbf','55a5a1ebe4b013909087cb79',
'55a5a1ebe4b013909087cbb0','55a5a1ebe4b013909087cbb3','55a5a1ebe4b013909087cb74',
'55a5a1ebe4b013909087cbaa','55a5a1ebe4b013909087cb83','55a5a1ebe4b013909087cb8c',
'55a5a1ebe4b013909087cb92','55a5a1ebe4b013909087cb8f','55a5a1ebe4b013909087cb86',
'55a5a1ebe4b013909087cbb9','55a5a1ebe4b013909087cb7f','55a5a1ebe4b013909087cbbc',
'55a5a1ebe4b013909087cb9e','55a5a1ebe4b013909087cbc2','55a5a1ebe4b013909087cbad']
def is_restaurant(categories, specific_filter=None):
restaurant_words = ['restaurant', 'diner', 'taverna', 'steakhouse']
restaurant = False
specific = False
for c in categories:
category_name = c[0].lower()
category_id = c[1]
for r in restaurant_words:
if r in category_name:
restaurant = True
if 'fast food' in category_name:
restaurant = False
if not(specific_filter is None) and (category_id in specific_filter):
specific = True
restaurant = True
return restaurant, specific
def get_categories(categories):
return [(cat['name'], cat['id']) for cat in categories]
def format_address(location):
address = ', '.join(location['formattedAddress'])
address = address.replace(', New York', '')
address = address.replace(', United States of America', '')
return address
def get_venues_near_location(lat, lon, category, CLIENT_ID, CLIENT_SECRET, radius=500, limit=100):
version = '20190720'
url = 'https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'.format(
CLIENT_ID, CLIENT_SECRET, version, lat, lon, category, radius, limit)
try:
results = requests.get(url).json()['response']['groups'][0]['items']
venues = [(item['venue']['id'],
item['venue']['name'],
get_categories(item['venue']['categories']),
(item['venue']['location']['lat'], item['venue']['location']['lng']),
format_address(item['venue']['location']),
item['venue']['location']['distance']) for item in results]
except:
venues = []
return venues
# Let's now go over our neighborhood locations and get nearby restaurants; we'll also maintain a dictionary of all found restaurants and all found italian restaurants
import pickle
def get_restaurants(lats, lons):
restaurants = {}
italian_restaurants = {}
location_restaurants = []
print('Obtaining venues around candidate locations:', end='')
for lat, lon in zip(lats, lons):
# Using radius=350 to meke sure we have overlaps/full coverage so we don't miss any restaurant (we're using dictionaries to remove any duplicates resulting from area overlaps)
venues = get_venues_near_location(lat, lon, food_category, CLIENT_ID, CLIENT_SECRET, radius=350, limit=100)
area_restaurants = []
for venue in venues:
venue_id = venue[0]
venue_name = venue[1]
venue_categories = venue[2]
venue_latlon = venue[3]
venue_address = venue[4]
venue_distance = venue[5]
is_res, is_italian = is_restaurant(venue_categories, specific_filter=italian_restaurant_categories)
if is_res:
x, y = lonlat_to_xy(venue_latlon[1], venue_latlon[0])
restaurant = (venue_id, venue_name, venue_latlon[0], venue_latlon[1], venue_address, venue_distance, is_italian, x, y)
if venue_distance<=300:
area_restaurants.append(restaurant)
restaurants[venue_id] = restaurant
if is_italian:
italian_restaurants[venue_id] = restaurant
location_restaurants.append(area_restaurants)
print(' .', end='')
print(' done.')
return restaurants, italian_restaurants, location_restaurants
# Try to load from local file system in case we did this before
restaurants = {}
italian_restaurants = {}
location_restaurants = []
loaded = False
try:
with open('restaurants_300.pkl', 'rb') as f:
restaurants = pickle.load(f)
with open('italian_restaurants_300.pkl', 'rb') as f:
italian_restaurants = pickle.load(f)
with open('location_restaurants_300.pkl', 'rb') as f:
location_restaurants = pickle.load(f)
print('Restaurant data loaded.')
loaded = True
except:
pass
# If load failed use the Foursquare API to get the data
if not loaded:
restaurants, italian_restaurants, location_restaurants = get_restaurants(latitudes, longitudes)
# Let's persists this in local file system
with open('restaurants_300.pkl', 'wb') as f:
pickle.dump(restaurants, f)
with open('italian_restaurants_300.pkl', 'wb') as f:
pickle.dump(italian_restaurants, f)
with open('location_restaurants_300.pkl', 'wb') as f:
pickle.dump(location_restaurants, f)
Obtaining venues around candidate locations: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . done.
import numpy as np
print('Total number of restaurants:', len(restaurants))
print('Total number of Italian restaurants:', len(italian_restaurants))
print('Percentage of Italian restaurants: {:.2f}%'.format(len(italian_restaurants) / len(restaurants) * 100))
print('Average number of restaurants in neighborhood:', np.array([len(r) for r in location_restaurants]).mean())
Total number of restaurants: 2287 Total number of Italian restaurants: 345 Percentage of Italian restaurants: 15.09% Average number of restaurants in neighborhood: 13.372670807453416
print('List of all restaurants')
print('-----------------------')
for r in list(restaurants.values())[:10]:
print(r)
print('...')
print('Total:', len(restaurants))
List of all restaurants
-----------------------
('4b527dc6f964a520ac7f27e3', 'LIC Market', 40.74749477775138, -73.94541675182658, '2152 44th Dr (at 23rd St), Long Island City, NY 11101, United States', 129, False, -5816181.640886299, 9861947.85684938)
('5c9a607d25fb7b002c724043', "Xi'an Famous Foods", 40.747488830456156, -73.94164499378402, '26-19 Jackson Ave, Long Island City, NY 11101, United States', 188, False, -5816168.981262614, 9861461.558703868)
('5a072da8b1538e5613afe96e', 'Indie', 40.74889496653177, -73.94216174033207, '43-10 Crescent St (43 Ave), NY 11101, United States', 208, False, -5815932.492408007, 9861534.88797734)
('53ed8f36498ea1eca81563c1', 'Il Falco', 40.74756750146132, -73.94539767076132, '2150 44th Dr, Long Island City, NY 11101, United States', 127, True, -5816169.243735234, 9861945.742738305)
('57f54617498e2cb89939bbfc', 'Chipotle Mexican Grill', 40.747227491157524, -73.94164881692888, '2616 Jackson Ave, Long Island City, NY 11101, United States', 191, False, -5816213.2973580975, 9861460.804152802)
('521d617c11d2d3f4edb81926', 'M. Wells Steakhouse', 40.74839103721746, -73.94233553952344, '43-15 Crescent St (43rd Ave), Long Island City, NY 11101, United States', 160, False, -5816018.546876245, 9861554.890004119)
('49ea8fb5f964a5207f661fe3', 'Court Square Diner', 40.74608787818452, -73.94561784321463, '45-30 23rd St (at 45th Rd), Long Island City, NY 11101, United States', 219, False, -5816420.869500532, 9861967.09188388)
('50574d6de4b0e7a978fa06b1', 'The Inkan', 40.746841, -73.945625, '4502 23rd St (at 45th Ave.), Long Island City, NY 11101, United States', 166, False, -5816293.223384095, 9861971.596105423)
('55d7b438498e926c55f30226', 'Bia', 40.7459817543423, -73.94540264221281, '23-10 Jackson Ave (at Pearson St), Long Island City, NY 11101, United States', 216, False, -5816438.081726819, 9861938.841113053)
('4a771fc8f964a520a9e31fe3', 'Gaw Gai', 40.745980187701825, -73.9455198305912, '23-06 Jackson Ave (at 23rd St), Long Island City, NY 11101, United States', 222, False, -5816438.771230165, 9861953.942877747)
...
Total: 2287
print('List of Italian restaurants')
print('---------------------------')
for r in list(italian_restaurants.values())[:10]:
print(r)
print('...')
print('Total:', len(italian_restaurants))
List of Italian restaurants
---------------------------
('53ed8f36498ea1eca81563c1', 'Il Falco', 40.74756750146132, -73.94539767076132, '2150 44th Dr, Long Island City, NY 11101, United States', 127, True, -5816169.243735234, 9861945.742738305)
('45912d3ef964a52029401fe3', 'Brooks 1890', 40.74653831124381, -73.9440769553404, '2428 Jackson Ave (Court Square), Long Island City, NY 11101, United States', 114, True, -5816338.933099656, 9861770.569738343)
('463220f3f964a520fb451fe3', "Manetta's Ristorante", 40.742577, -73.95194, '1076 Jackson Ave (btwn Hunters Point Ave. & 11th St.), Long Island City, NY 11101, United States', 155, True, -5817038.873882278, 9862765.616473733)
('4f74ff83e4b09b101271b89b', 'Manducatis', 40.7445680769774, -73.9490078387153, '13-27 Jackson Ave, Long Island City, NY 11101, United States', 261, True, -5816690.76005959, 9862396.962125987)
('49f5bf5ef964a520b46b1fe3', 'Manducatis Rustica', 40.745959142740666, -73.95302063397736, '4635 Vernon Blvd (btwn 46th Rd. & 47th Ave.), Long Island City, NY 11101, United States', 322, True, -5816469.374472322, 9862920.933982342)
('4e4078922271a89dd083d07e', 'Felice 83', 40.77486666568879, -73.950868085548, '1593 1st Ave (at E 83rd St.), NY 10028, United States', 177, True, -5811562.447725797, 9862780.113703828)
('504bb0fae4b0d1cf33ca07d0', '83 1/2', 40.77520589574406, -73.95123198330161, '345 E 83rd St (at 1st Ave), NY 10028, United States', 331, True, -5811506.2796454765, 9862828.587215)
('4b623700f964a520a83c2ae3', 'nica trattoria', 40.775688458031425, -73.95057000240742, '354 E 84th St, NY 10028, United States', 213, True, -5811422.141813967, 9862745.605403755)
('4dcf3f00d22deadedd682925', 'Luna Rossa', 40.77654404044222, -73.95032631591812, '347 E 85th St (btw 1st & 2nd), NY 10028, United States', 284, True, -5811276.30866058, 9862718.264956096)
('4ad52027f964a5200a0221e3', 'Ginos Pizzeria', 40.775231503163006, -73.9513015269503, '345 E 83rd St (at 1st Ave), NY 10028, United States', 228, True, -5811502.190805928, 9862837.66544024)
...
Total: 345
print('Restaurants around location')
print('---------------------------')
for i in range(95, 110):
rs = location_restaurants[i][:8]
names = ', '.join([r[1] for r in rs])
print('Restaurants around location {}: {}'.format(i+1, names))
Restaurants around location --------------------------- Restaurants around location 96: Le Soleil Haitian Restaurant, The Greek Kitchen Restaurants around location 97: La Salle Dumpling Room, Cafe 21, Mah Jong Restaurants around location 98: Pier I Cafe Restaurants around location 99: Restaurants around location 100: Restaurants around location 101: Carbone, Tomoe Sushi, Lupa, ZZ's Clam Bar, Cuba, Junzi Kitchen (Greenwich Village), DomoDomo, Carroll Place Restaurants around location 102: Loring Place, Blue Hill, Alta, Babbo Ristorante e Enoteca, Eva's Kitchen, Quality Eats, Omakase Room by Tatsu, Rosemary’s Restaurants around location 103: Coppelia, Bar B, Flex Mussels, The James Beard House, Sotto 13, Green Symphony, Gradisca, Noodle King Of NY Restaurants around location 104: Momoya, Westville Chelsea, Foragers Table, Socarrat Paella Bar, Momofuku Nishi, El Cocotero, Le Zie 2000 Trattoria, Sushi Seki Chelsea Restaurants around location 105: Jun-Men Ramen Bar, Chelsea Fine Foods, Txikito, Grand Sichuan International, Riko Peruvian Cuisine, Purple Rice, biricchino, Songkran Thai Kitchen Restaurants around location 106: Sergimmo Salumeria, Friedman's, Hudson Yards Grill, Kawi, Chef's Table At Brooklyn Fare, Uncle Jack's Steakhouse, Zoob Zib Thai Noodle Bar, Thai Select Restaurants around location 107: La Vela Dining & Bar, Legacy Records, Treadwell, West Side Steakhouse, East & West, EDEN Local, Green Fig, Poseidon Bakery Restaurants around location 108: Print, dell’anima, Genuine Roadside, Rooftop 48, Jianbing Company, Seamore’s, Ivan Ramen Slurp Shop, Cantina Rooftop Restaurants around location 109: Pier 95 Restaurants around location 110: Bistro 61
Let's now see all the collected restaurants in our area of interest on map, and let's also show Italian restaurants in different color.
map_ny_city = folium.Map(location=ny_city, zoom_start=13)
folium.Marker(ny_city, popup='Times Square').add_to(map_ny_city)
for res in restaurants.values():
lat = res[2]; lon = res[3]
is_italian = res[6]
color = 'blue' if is_italian else 'red'
folium.CircleMarker([lat, lon], radius=3, color=color, fill=True, fill_color=color, fill_opacity=1).add_to(map_ny_city)
map_ny_city
Looking good. So now we have all the restaurants in area within few kilometers from Times Square, NY, and we know which ones are Italian restaurants! We also know which restaurants exactly are in vicinity of every neighborhood candidate center.
This concludes the data gathering phase - we're now ready to use this data for analysis to produce the report on optimal locations for a new Italian restaurant!
In this project we will direct our efforts on detecting areas of Berlin that have low restaurant density, particularly those with low number of Italian restaurants. We will limit our analysis to area ~6km around city center.
In first step we have collected the required data: location and type (category) of every restaurant within 6km from Times Square (New York City). We have also identified Italian restaurants (according to Foursquare categorization).
Second step in our analysis will be calculation and exploration of 'restaurant density' across different areas of Berlin - we will use heatmaps to identify a few promising areas close to center with low number of restaurants in general (and no Italian restaurants in vicinity) and focus our attention on those areas.
In third and final step we will focus on most promising areas and within those create clusters of locations that meet some basic requirements established in discussion with stakeholders: we will take into consideration locations with no more than two restaurants in radius of 250 meters, and we want locations without Italian restaurants in radius of 400 meters. We will present map of all such locations but also create clusters (using k-means clustering) of those locations to identify general zones / neighborhoods / addresses which should be a starting point for final 'street level' exploration and search for optimal venue location by stakeholders.
Let's perform some basic explanatory data analysis and derive some additional info from our raw data. First let's count the number of restaurants in every area candidate:
location_restaurants_count = [len(res) for res in location_restaurants]
df_locations['Restaurants in area'] = location_restaurants_count
print('Average number of restaurants in every area with radius=300m:', np.array(location_restaurants_count).mean())
df_locations.head(10)
Average number of restaurants in every area with radius=300m: 13.372670807453416
| Address | Latitude | Longitude | X | Y | Distance from center | Restaurants in area | |
|---|---|---|---|---|---|---|---|
| 0 | 25-20 49th Avenue, New York, NY 11101 | 40.742248 | -73.944078 | -5.817066e+06 | 9.861750e+06 | 5960.494946 | 0 |
| 1 | Two Court Square, 23-21 44th Drive, New York, ... | 40.747552 | -73.943882 | -5.816166e+06 | 9.861750e+06 | 5658.400834 | 15 |
| 2 | Citi Bike - 21 St & Queens Plaza North, 21st S... | 40.752858 | -73.943686 | -5.815266e+06 | 9.861750e+06 | 5488.852339 | 1 |
| 3 | 38-36 10th Street, New York, NY 11101 | 40.758163 | -73.943490 | -5.814366e+06 | 9.861750e+06 | 5464.201680 | 3 |
| 4 | 36th Avenue, New York, NY 11106 | 40.763470 | -73.943294 | -5.813466e+06 | 9.861750e+06 | 5586.367335 | 2 |
| 5 | Octagon Tennis Courts, Road 10, New York, NY 1... | 40.768777 | -73.943097 | -5.812566e+06 | 9.861750e+06 | 5846.152581 | 0 |
| 6 | 43 Paidge Avenue, New York, NY 11222 | 40.737072 | -73.950312 | -5.817966e+06 | 9.862530e+06 | 5723.635209 | 0 |
| 7 | 11-39 50th Avenue, New York, NY 11101 | 40.742376 | -73.950118 | -5.817066e+06 | 9.862530e+06 | 5256.424640 | 12 |
| 8 | 45-02 11th Street, New York, NY 11101 | 40.747681 | -73.949923 | -5.816166e+06 | 9.862530e+06 | 4911.211663 | 4 |
| 9 | 8-05 43rd Avenue, New York, NY 11101 | 40.752986 | -73.949728 | -5.815266e+06 | 9.862530e+06 | 4714.870094 | 1 |
OK, now let's calculate the distance to nearest Italian restaurant from every area candidate center (not only those within 300m - we want distance to closest one, regardless of how distant it is).
distances_to_italian_restaurant = []
for area_x, area_y in zip(xs, ys):
min_distance = 10000
for res in italian_restaurants.values():
res_x = res[7]
res_y = res[8]
d = calc_xy_distance(area_x, area_y, res_x, res_y)
if d<min_distance:
min_distance = d
distances_to_italian_restaurant.append(min_distance)
df_locations['Distance to Italian restaurant'] = distances_to_italian_restaurant
df_locations.head(10)
| Address | Latitude | Longitude | X | Y | Distance from center | Restaurants in area | Distance to Italian restaurant | |
|---|---|---|---|---|---|---|---|---|
| 0 | 25-20 49th Avenue, New York, NY 11101 | 40.742248 | -73.944078 | -5.817066e+06 | 9.861750e+06 | 5960.494946 | 0 | 727.663490 |
| 1 | Two Court Square, 23-21 44th Drive, New York, ... | 40.747552 | -73.943882 | -5.816166e+06 | 9.861750e+06 | 5658.400834 | 15 | 173.813115 |
| 2 | Citi Bike - 21 St & Queens Plaza North, 21st S... | 40.752858 | -73.943686 | -5.815266e+06 | 9.861750e+06 | 5488.852339 | 1 | 923.853177 |
| 3 | 38-36 10th Street, New York, NY 11101 | 40.758163 | -73.943490 | -5.814366e+06 | 9.861750e+06 | 5464.201680 | 3 | 1813.499568 |
| 4 | 36th Avenue, New York, NY 11106 | 40.763470 | -73.943294 | -5.813466e+06 | 9.861750e+06 | 5586.367335 | 2 | 1459.955101 |
| 5 | Octagon Tennis Courts, Road 10, New York, NY 1... | 40.768777 | -73.943097 | -5.812566e+06 | 9.861750e+06 | 5846.152581 | 0 | 1255.680177 |
| 6 | 43 Paidge Avenue, New York, NY 11222 | 40.737072 | -73.950312 | -5.817966e+06 | 9.862530e+06 | 5723.635209 | 0 | 658.914606 |
| 7 | 11-39 50th Avenue, New York, NY 11101 | 40.742376 | -73.950118 | -5.817066e+06 | 9.862530e+06 | 5256.424640 | 12 | 237.540592 |
| 8 | 45-02 11th Street, New York, NY 11101 | 40.747681 | -73.949923 | -5.816166e+06 | 9.862530e+06 | 4911.211663 | 4 | 494.911255 |
| 9 | 8-05 43rd Avenue, New York, NY 11101 | 40.752986 | -73.949728 | -5.815266e+06 | 9.862530e+06 | 4714.870094 | 1 | 1075.291078 |
print('Average distance to closest Italian restaurant from each area center:', df_locations['Distance to Italian restaurant'].mean())
Average distance to closest Italian restaurant from each area center: 619.6807355025484
OK, so on average Italian restaurant can be found within ~1500m from every area center candidate. That's fairly close, so we need to filter our areas carefully!
Let's crete a map showing heatmap / density of restaurants and try to extract some meaningfull info from that. Also, let's show 49th Avenue and43 Paidge Avenue, NY on our map and a few circles indicating distance of 1km, 2km and 3km from Times Square.
!wget -q -O 'newyork_data.json' https://cocl.us/new_york_dataset
print('Data downloaded!')
Data downloaded!
with open('newyork_data.json') as json_data:
newyork_data = json.load(json_data)
def boroughs_style(feature):
return { 'color': 'blue', 'fill': False }
restaurant_latlons = [[res[2], res[3]] for res in restaurants.values()]
italian_latlons = [[res[2], res[3]] for res in italian_restaurants.values()]
from folium import plugins
from folium.plugins import HeatMap
map_ny_city = folium.Map(location=ny_city, zoom_start=13)
folium.TileLayer('cartodbpositron').add_to(map_ny_city) #cartodbpositron cartodbdark_matter
HeatMap(restaurant_latlons).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
folium.Circle(ny_city, radius=1000, fill=False, color='white').add_to(map_ny_city)
folium.Circle(ny_city, radius=2000, fill=False, color='white').add_to(map_ny_city)
folium.Circle(ny_city, radius=3000, fill=False, color='white').add_to(map_ny_city)
folium.Circle(ny_city, radius=4000, fill=False, color='white').add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='Geojson').add_to(map_ny_city)
map_ny_city
Looks like a few pockets of low restaurant density closest to city center can be found south-east and east from Times Square.
Let's create another heatmap map showing heatmap/density of Italian restaurants only.
map_ny_city = folium.Map(location=ny_city, zoom_start=13)
folium.TileLayer('cartodbpositron').add_to(map_ny_city) #cartodbpositron cartodbdark_matter
HeatMap(italian_latlons).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
folium.Circle(ny_city, radius=1000, fill=False, color='white').add_to(map_ny_city)
folium.Circle(ny_city, radius=2000, fill=False, color='white').add_to(map_ny_city)
folium.Circle(ny_city, radius=3000, fill=False, color='white').add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
This map is not so 'hot' (Italian restaurants represent a subset of ~15% of all restaurants in New York) but it also indicates higher density of existing Italian restaurants directly north and west from Times Square, with closest pockets of low Italian restaurant density positioned east, south-east from city center.
Based on this we will now focus our analysis on areas south-west, south, south-east and east from New York - we will move the center of our area of interest and reduce it's size to have a radius of 2.5km. This places our location candidates mostly in boroughs MidTown Manhattan and Murray Hill (another potentially interesting borough is Long Inland City with large low restaurant density north-east from city center, however this borough is less interesting to stakeholders as it's mostly residental and less popular with tourists).
Analysis of popular travel guides and web sites often mention Times Square and Museumes and Therters as beautifull, interesting, rich with culture, 'hip' and 'cool' New York neighborhoods popular with tourists and loved by New Yorkers.
"Bold and brazen, Brooklyn 's creative people, places, and spaces might challenge your paradigm." Tags: Nightlife, Artsy, Dining, Trendy, Loved by New Yorkers.
Midtown Manhattan is the central portion of the borough of Manhattan in New York City. Midtown is home to some of the city's most iconic buildings, including the Empire State Building, the Chrysler Building, the Hudson Yards Redevelopment Project, the headquarters of the United Nations, Grand Central Terminal, and Rockefeller Center, as well as Broadway and Times Square.
Midtown Manhattan is the largest central business district in the world and ranks among the most expensive pieces of real estate; Fifth Avenue in Midtown Manhattan commands the world's highest retail rents, with average annual rents at US$3,000 per square foot ($32,000/m2) in 2017.[4] However, due to the high price of retail spaces in Midtown, there are also many vacant storefronts in the neighborhood.[5] Midtown is the country's largest commercial, entertainment, and media center, and also a growing financial center.
Murray Hill is a neighborhood on the east side of Manhattan in New York City. Murray Hill is bordered to the east by the East River, to the west by Midtown Manhattan, to the south by Kips Bay and Rose Hill, and to the north by Turtle Bay. Its exact boundaries are disputed and vary widely, but it is generally located between East 32nd Street and/or 34th Street to the south, East 40th Street and/or 42nd Street to the north, Madison Avenue or Fifth Avenue to the west, and the East River to the east.
Let's define new, more narrow region of interest, which will include low-restaurant-count parts of closest to Times Square.
roi_x_min = ny_city_x - 2000
roi_y_max = ny_city_y + 1000
roi_width = 5000
roi_height = 5000
roi_center_x = roi_x_min + 2500
roi_center_y = roi_y_max - 2500
roi_center_lon, roi_center_lat = xy_to_lonlat(roi_center_x, roi_center_y)
roi_center = [roi_center_lat, roi_center_lon]
map_ny_city = folium.Map(location=roi_center, zoom_start=14)
HeatMap(restaurant_latlons).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
folium.Circle(roi_center, radius=2500, color='white', fill=True, fill_opacity=0.4).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Not bad - this nicely covers all the pockets of low restaurant density in MidTown Manhattan and Murray Hill closest to Times Square.
Let's also create new, more dense grid of location candidates restricted to our new region of interest (let's make our location candidates 100m appart).
k = math.sqrt(3) / 2 # Vertical offset for hexagonal grid cells
x_step = 100
y_step = 100 * k
roi_y_min = roi_center_y - 2500
roi_latitudes = []
roi_longitudes = []
roi_xs = []
roi_ys = []
for i in range(0, int(51/k)):
y = roi_y_min + i * y_step
x_offset = 50 if i%2==0 else 0
for j in range(0, 51):
x = roi_x_min + j * x_step + x_offset
d = calc_xy_distance(roi_center_x, roi_center_y, x, y)
if (d <= 2501):
lon, lat = xy_to_lonlat(x, y)
roi_latitudes.append(lat)
roi_longitudes.append(lon)
roi_xs.append(x)
roi_ys.append(y)
print(len(roi_latitudes), 'candidate neighborhood centers generated.')
2261 candidate neighborhood centers generated.
OK. Now let's calculate two most important things for each location candidate: number of restaurants in vicinity (we'll use radius of 250 meters) and distance to closest Italian restaurant.
def count_restaurants_nearby(x, y, restaurants, radius=250):
count = 0
for res in restaurants.values():
res_x = res[7]; res_y = res[8]
d = calc_xy_distance(x, y, res_x, res_y)
if d<=radius:
count += 1
return count
def find_nearest_restaurant(x, y, restaurants):
d_min = 100000
for res in restaurants.values():
res_x = res[7]; res_y = res[8]
d = calc_xy_distance(x, y, res_x, res_y)
if d<=d_min:
d_min = d
return d_min
roi_restaurant_counts = []
roi_italian_distances = []
print('Generating data on location candidates... ', end='')
for x, y in zip(roi_xs, roi_ys):
count = count_restaurants_nearby(x, y, restaurants, radius=250)
roi_restaurant_counts.append(count)
distance = find_nearest_restaurant(x, y, italian_restaurants)
roi_italian_distances.append(distance)
print('done.')
Generating data on location candidates... done.
# Let's put this into dataframe
df_roi_locations = pd.DataFrame({'Latitude':roi_latitudes,
'Longitude':roi_longitudes,
'X':roi_xs,
'Y':roi_ys,
'Restaurants nearby':roi_restaurant_counts,
'Distance to Italian restaurant':roi_italian_distances})
df_roi_locations.head(10)
| Latitude | Longitude | X | Y | Restaurants nearby | Distance to Italian restaurant | |
|---|---|---|---|---|---|---|
| 0 | 40.759288 | -73.954746 | -5.814216e+06 | 9.863206e+06 | 0 | 862.458710 |
| 1 | 40.759877 | -73.954724 | -5.814116e+06 | 9.863206e+06 | 0 | 841.627882 |
| 2 | 40.756059 | -73.955536 | -5.814766e+06 | 9.863293e+06 | 0 | 1077.023383 |
| 3 | 40.756649 | -73.955514 | -5.814666e+06 | 9.863293e+06 | 0 | 1007.214890 |
| 4 | 40.757238 | -73.955493 | -5.814566e+06 | 9.863293e+06 | 0 | 942.859640 |
| 5 | 40.757828 | -73.955471 | -5.814466e+06 | 9.863293e+06 | 0 | 885.147878 |
| 6 | 40.758418 | -73.955450 | -5.814366e+06 | 9.863293e+06 | 0 | 835.457499 |
| 7 | 40.759007 | -73.955428 | -5.814266e+06 | 9.863293e+06 | 0 | 795.293467 |
| 8 | 40.759597 | -73.955407 | -5.814166e+06 | 9.863293e+06 | 0 | 766.155444 |
| 9 | 40.760186 | -73.955385 | -5.814066e+06 | 9.863293e+06 | 0 | 749.330789 |
OK. Let us now filter those locations: we're interested only in locations with no more than two restaurants in radius of 250 meters, and no Italian restaurants in radius of 400 meters.
good_res_count = np.array((df_roi_locations['Restaurants nearby']<=2))
print('Locations with no more than two restaurants nearby:', good_res_count.sum())
good_ita_distance = np.array(df_roi_locations['Distance to Italian restaurant']>=400)
print('Locations with no Italian restaurants within 400m:', good_ita_distance.sum())
good_locations = np.logical_and(good_res_count, good_ita_distance)
print('Locations with both conditions met:', good_locations.sum())
df_good_locations = df_roi_locations[good_locations]
Locations with no more than two restaurants nearby: 668 Locations with no Italian restaurants within 400m: 654 Locations with both conditions met: 465
Let's see how this looks on a map.
good_latitudes = df_good_locations['Latitude'].values
good_longitudes = df_good_locations['Longitude'].values
good_locations = [[lat, lon] for lat, lon in zip(good_latitudes, good_longitudes)]
map_berlin = folium.Map(location=roi_center, zoom_start=14)
folium.TileLayer('cartodbpositron').add_to(map_ny_city)
HeatMap(restaurant_latlons).add_to(map_ny_city)
folium.Circle(roi_center, radius=2500, color='white', fill=True, fill_opacity=0.6).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Looking good. We now have a bunch of locations fairly close to Times Square, and we know that each of those locations has no more than two restaurants in radius of 250m, and no Italian restaurant closer than 400m. Any of those locations is a potential candidate for a new Italian restaurant, at least based on nearby competition.
Let's now show those good locations in a form of heatmap:
map_ny_city = folium.Map(location=roi_center, zoom_start=14)
HeatMap(good_locations, radius=25).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Looking good. What we have now is a clear indication of zones with low number of restaurants in vicinity, and no Italian restaurants at all nearby.
Let us now cluster those locations to create centers of zones containing good locations. Those zones, their centers and addresses will be the final result of our analysis.
from sklearn.cluster import KMeans
number_of_clusters = 15
good_xys = df_good_locations[['X', 'Y']].values
kmeans = KMeans(n_clusters=number_of_clusters, random_state=0).fit(good_xys)
cluster_centers = [xy_to_lonlat(cc[0], cc[1]) for cc in kmeans.cluster_centers_]
map_ny_city = folium.Map(location=roi_center, zoom_start=14)
folium.TileLayer('cartodbpositron').add_to(map_ny_city)
HeatMap(restaurant_latlons).add_to(map_ny_city)
folium.Circle(roi_center, radius=2500, color='white', fill=True, fill_opacity=0.4).add_to(map_ny_city)
folium.Marker(ny_city).add_to(map_ny_city)
for lon, lat in cluster_centers:
folium.Circle([lat, lon], radius=500, color='green', fill=True, fill_opacity=0.25).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Not bad - our clusters represent groupings of most of the candidate locations and cluster centers are placed nicely in the middle of the zones 'rich' with location candidates.
Addresses of those cluster centers will be a good starting point for exploring the neighborhoods to find the best possible location based on neighborhood specifics.
Let's see those zones on a city map without heatmap, using shaded areas to indicate our clusters:
map_ny_city= folium.Map(location=roi_center, zoom_start=14)
folium.Marker(ny_city).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.Circle([lat, lon], radius=250, color='#00000000', fill=True, fill_color='#0066ff', fill_opacity=0.07).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
for lon, lat in cluster_centers:
folium.Circle([lat, lon], radius=500, color='green', fill=False).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Let's zoom in on candidate areas in MidTown Manhattan:
map_ny_city= folium.Map(location=[40.7622684, -73.9795443], zoom_start=15)
folium.Marker(ny_city).add_to(map_ny_city)
for lon, lat in cluster_centers:
folium.Circle([lat, lon], radius=500, color='green', fill=False).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.Circle([lat, lon], radius=250, color='#0000ff00', fill=True, fill_color='#0066ff', fill_opacity=0.07).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
...and candidate areas in Murray Hill:
map_ny_city = folium.Map(location=[40.7622684, -73.9795443], zoom_start=15)
folium.Marker(ny_city).add_to(map_ny_city)
for lon, lat in cluster_centers:
folium.Circle([lat, lon], radius=500, color='green', fill=False).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.Circle([lat, lon], radius=250, color='#0000ff00', fill=True, fill_color='#0066ff', fill_opacity=0.07).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.CircleMarker([lat, lon], radius=2, color='blue', fill=True, fill_color='blue', fill_opacity=1).add_to(map_ny_city)
folium.GeoJson(newyork_data, style_function=boroughs_style, name='geojson').add_to(map_ny_city)
map_ny_city
Finaly, let's reverse geocode those candidate area centers to get the addresses which can be presented to stakeholders.
candidate_area_addresses = []
print('==============================================================')
print('Addresses of centers of areas recommended for further analysis')
print('==============================================================\n')
for lon, lat in cluster_centers:
addr = get_address(OpenCage_api_key, lat, lon).replace(', United States of America', '')
candidate_area_addresses.append(addr)
x, y = lonlat_to_xy(lon, lat)
d = calc_xy_distance(x, y, ny_city_x, ny_city_y)
print('{}{} => {:.1f}km from Times Square'.format(addr, ' '*(50-len(addr)), d/1000))
============================================================== Addresses of centers of areas recommended for further analysis ============================================================== Umpire Rock, Pinebank Arch, New York, NY 10106 => 2.3km from Times Square 501 East 56th Street, New York, NY 10022 => 3.3km from Times Square Rumsey Playfield, 72nd Street Transverse, New York, NY 10065 => 3.3km from Times Square United Nations Headquarters, 405 FDR Drive, New York, NY 10017 => 2.7km from Times Square Rockefeller University, 1230 York Avenue, New York, NY 10065 => 3.9km from Times Square Mount Sinai West, 1000 10th Avenue, New York, NY 10019 => 2.1km from Times Square Roosevelt Island Greenway, New York, NY 10155 => 3.6km from Times Square Ed Koch Queensboro Bridge, Ed Koch Queensboro Bridge Upper Level, New York, NY 10155 => 3.7km from Times Square 6 East 68th Street, New York, NY 10065 => 3.1km from Times Square Center Drive, New York, NY 10153 => 2.6km from Times Square FDR Drive, New York, NY 10017-6927 => 3.1km from Times Square 340 Madison Avenue, New York, NY 10017 => 1.0km from Times Square 2 72nd Street Transverse, New York, NY 10023 => 3.0km from Times Square FDR Drive, New York, NY 10017 => 2.7km from Times Square New York, NY => 3.3km from Times Square
This concludes our analysis. We have created 15 addresses representing centers of zones containing locations with low number of restaurants and no Italian restaurants nearby, all zones being fairly close to city center (all less than 4km from Times Square. Although zones are shown on map with a radius of ~500 meters (green circles), their shape is actually very irregular and their centers/addresses should be considered only as a starting point for exploring area neighborhoods in search for potential restaurant locations. Most of the zones are located in Kreuzberg and Friedrichshain boroughs, which we have identified as interesting due to being popular with tourists, fairly close to city center and well connected by public transport.
map_ny_city = folium.Map(location=roi_center, zoom_start=14)
folium.Circle(ny_city, radius=50, color='red', fill=True, fill_color='red', fill_opacity=1).add_to(map_ny_city)
for lonlat, addr in zip(cluster_centers, candidate_area_addresses):
folium.Marker([lonlat[1], lonlat[0]], popup=addr).add_to(map_ny_city)
for lat, lon in zip(good_latitudes, good_longitudes):
folium.Circle([lat, lon], radius=250, color='#0000ff00', fill=True, fill_color='#0066ff', fill_opacity=0.05).add_to(map_ny_city)
map_ny_city
Our analysis shows that although there is a great number of restaurants in New York (~2300 in our initial area of interest which was 12x12km around Times Square, NY), there are pockets of low restaurant density fairly close to city centre. Highest concentration of restaurants was detected north and west from Times Square, so we focused our attention to areas south, south-east and east, corresponding to boroughs MidTown Manhattan and Murray Hill and south-east. After directing our attention to this more narrow area of interest (covering approx. 5x5km south-east from Times Square, NY) we first created a dense grid of location candidates (spaced 100m appart); those locations were then filtered so that those with more than two restaurants in radius of 250m and those with an Italian restaurant closer than 400m were removed.
Those location candidates were then clustered to create zones of interest which contain greatest number of location candidates. Addresses of centres of those zones were also generated using reverse geocoding to be used as markers/starting points for more detailed local analysis based on other factors.
Result of all these 15 zones containing largest number of potential new restaurant locations based on the distance to existing venues - both restaurants in general and Italian restaurants particularly. This, of course, does not imply that those zones are optimal locations for a new restaurant! Purpose of this analysis was to only provide info on areas close to Times Square but not crowded with existing restaurants (particularly Italian) - it is entirely possible that there is a very good reason for small number of restaurants in any of those areas, reasons which would make them unsuitable for a new restaurant regardless of lack of competition in the area. Recommended zones should therefore be considered only as a starting point for more detailed analysis which could eventually result in location which has not only no nearby competition, but also other factors taken into account and all other relevant conditions met.
Purpose of this project was to identify New York areas close to centre with low number of restaurants (particularly Italian restaurants) in order to aid stakeholders in narrowing down the search for optimal location for a new Italian restaurant. By calculating restaurant density distribution from Foursquare data we have first identified general areas that justify further analysis (MidTown Manhattan and Murray Hill), and then generated extensive collection of locations which satisfy some basic requirements regarding existing nearby restaurants. Clustering of those locations was then performed in order to create major zones of interest (containing greatest number of potential locations) and addresses of those zone centres were created to be used as starting points for final exploration by stakeholders.
Final decission on optimal restaurant location will be made by stakeholders based on specific characteristics of neighborhoods and locations in every recommended zone, taking into consideration additional factors like attractiveness of each location (proximity to park or water), levels of noise / proximity to major roads, real estate availability, prices, social and economic dynamics of every neighborhood etc.